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ABSTRACT 

If school prifcipals are +0 exercise adequate 
tianadement control over thei educational prograus,. they must have | 
effective management tools. Title I Program Analysis and Monitoring 
s (PAM) 4s such a management tool.. Title I PAM is useful, timely, and 
effictent--in the sense of minimum interference with the educational 
prodram and modest cost. The Title I PAM System in reading comprises 
‘Achievement Monitors for the measurement Of reading comprehension:, 
Monitor Reports on teachers’ reading classes or instructional qrougs: 
Proaram Analysis Repofts (PAP): gauéstionnaires for the collection of ‘ 
process informatign: and comput Programs for test scoring and . 
teport generation. PAR (i.e.,: gfade-level reports for the principal) 
are the heart of PAM and relate pupil achievement to the distribution . 
and use of classroom resources. If pupil achievement is substandard, 
then the principal has a legitimate heed to intervene at the level of 
@he classtoom teacher. If pypil achievement is satisfactory, but 
resource utilization is more than, or less than, expected, the 
principal will want to confer:with the teacher. In Title I PAM there 
are three variations on the PAR. Descriptions and illustrations of 
each tvpe are appended. fRLt 
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Program Analysis and Monitoring (PAM) Techniques 
for Improving Title 1 Evaluation Loéally 


Steven Ls Kidder and Paul D. Hayford 


Problem and Perspective Needed. es) —~ “8 


Sustained school effectiveness does not,come easy. The education come 
munity continues to xeceive heavy blows. We are literally reeling from the 


effects’ of Heclining enrollment and achievement, and skyrocketing costs. This. 


‘is a tough time to be in the field of education to, “say i of the field 


of educational’ evdluation ox research. 


Have we helped our sghools? Have our perspectives and technical, approaches “ 
improved the situation at the classroom level? Has” our etphasis on curriculum 


devel opment, instructional practice, and testing Baid off? Oy Rave we for=| 
i 


gotten that teachers york with and for principals aud boards of ‘educapion 


and pret what teachers do in the classroom is legitimized by Lpcal cae “3 


educational poliey? “Havent t we forgotten the import ané¢ ‘of educational pros 


gram management, control; and direction? A fiveestar general seldom asks a 

private why 90% of the platoon the ‘private was in was destroyed. The private: 
~ * . oa eo al . - a ‘ 

is not accountable for such.a loss, but someone is + perhaps the ‘sergeant or 


Heutena tt or company commander should RS invetrogeced? Me 
Whae is taught in the classroom and how it: is taught should, be defined 
. 


by specific educational programs. - Thesé propank det theit Legitimacy 
from the Longaterm goals or mission of & a-school’ district, which, in turn, ? 


should reflect State criteria and curricula. This perspective on . , . 
id a eta 
oo management and program legitimacy is illustrated roughly, in. «: 


Figure te We beiidve” that'this perspective or(nanagement and jcontrol of 


instruction is essential to sustaining improvements in echo effectiveness. 


Figure 1 supports the conclusion that instructional effectiveness is a 


ee / ‘ 
function of program management capabilities, however loosely or tightly { 


4 
: 


‘ 


. expertise because ahey know nes to cont igure program resources (eB, staff, af 


. facilities, niterials,, instructional emphasis “and time) to maximize student . 


, within the definitional requhremanes ioe the Programe The teacher's basic 


thesé program management copepr anes are exercised. ~ ‘ % 


* there is a need to reconsider tastedctioaal Leadership and program 
management Capability ‘in, our schools. The need is great. And yets” sir 


are some good séhool (ptincipals who Continually get high Tevels of pevesauance’ 


- from, their teachers and students: at ‘ ‘ ve 


Good school princtpaly ag swell with teachers of” varied. backgrounds and 


achievenent Additionally, good principals explicitly or implicity exercise a 


basic program managetient model=-a model that is’ probably sinilay | to the one 


illustrated in Fighre 2. (This model reflects the ieerative nature of ‘Program | 


planning and <a based on evaluative data} that iis, by ihe. compattison 
a 
of ‘all types of pte emacs data to incerta’ program, startdgrds, including oe 


“staffing ‘Cnputs), reSource use (processes), and student’ achievement, Soutputs).« { 
‘this ‘basic program management model requires the definition “Oe Program standards; 
‘for inputs, processes, and outpugs while ‘Stuf tcaneoushy ‘ ‘sunpopttie, and 4 


< 
facilitating Anstruck ton Ber se se ‘in the nstioen) a? . i 8 
Good teachers sige. exercise a basic ‘instructional rangement model 


” anseructiongl model is iiserated in ‘Figure 3. te is anal ogous to the 


basic program management, mgdel of the principal and derives its, dixeck Lon 

and purpose ‘trom. progran definition, requirements, and Standards. Good . 
teachers have \specttic, cusedculumerelevant learning objectives, (Standards? 
hor all of their students. When evaluation data proyide ene students 


ate not ‘ achieving intended” "Learning oytcomes (standards), instructional activities 


are appropriately altered. g* times, even the direction ‘of the entire program 
: ; i 7 on . ‘ e 


must be changed. Nevertheless," good teachers handle themselves systematically 
: , : i oe : ae: 


. Le 


, within the guidelines ofitandards of. the inseruct, onal program: Thetz P “ee 


eee bo ‘ : 
\ ° approach usually, incorpogates the following mastery Yearning pristaples: . 
~~ 2 a * 
-" ao a) Learning units are ‘vequepcedy, (ay: students. are’ instructed in what is to’ 7 
= Z 
. ° Je, 4 


be learned and are, given dirgetions as" fo what’ they must do qurig & the 7 
learning process (cueing) » 43)" student's are rewarded: “or ieantorced for 
_ learning, (4) tive teacher e instricttonal ‘method engages the student- in overt ere 


and covert participation and. -response” to “deaeatie (motivation), ad 45) a or 


é s wah oe. . ys 


ie _frequent formative tests are used to, provide information in suppbrt ‘Of. better eo 


no “& orrective) “insticuctional sbeivitiegs” 


tro The principal is suggested ‘as the most. influential danaget and “dectsisn,- 


“oe : x ‘ 


. Maker in eddeation because ks the pervasive “effect he. has, cart ‘have, of - 


e . ace , 


‘pheals be- baving on teachets aad studants-. te he cannot manage Gress, ‘plan, 2 : 


util Atiprovetents in ‘stydede ypfobante 2 an! Rot ‘be ronpayiray . “paste , . 


a Rieti be we smplex learning environment that, a principal met manage 
fs . = r t 


there are other antl pétiios: ‘abe aia ‘aédels aa *ehoat Se eg 
* 9 Learning emdromme sits,’ the one obese tndludes- inberyeptonal variables. : ; 
praven ettects: on or positive ee O Hips’ t 


ray hese 


_ Rrincil 1 or ‘telcher can. ‘alter ps hang” tiem: esse ott aioe avaluat ive 


A 


feedbacks. le vents be 


“Tes was noted above. that peinctpals do 3 in 


they instructional progres and'yet bed “continue to be responsible for 
te : . . 
. be 


a 
*See "The State of Research. on setacced Alterable Variabfes-4h Ediic ation,'t 
MESA Seminar, Department of Education, University of Chicago, 19807 . 


fee . . foe: oy 2 : 
program operations ‘a accountable for prégram fuccess= aaéietony the 


“principal's reeponsibtldgy and accountability go well, peyond instruction’ 


into persénnel, bussings, and even £odd sgeviges In this couplex envitone 
x. 
: “ment it is “essential that _systeitatie approaches to the management of’ 


“ghstracetonal ptograms ‘be ‘ehcouraged, Good ' management of faserucei onay pros 


uses: ‘is‘a must.’ Instructional programs ‘must be defined clearly in basic’ 
me ; , 


ways before tational management and evaluation a effectiveness are possible, ' 
A. principal fust be able to specify allocated résources, basit instructional 
processes, and expected stident achievement . If instructional programs are. not 
defined, rational Program modification and itive .are not possible, 
However, “nseruct tonal programs can. be specified with definitions of inputs, 
processes, and outputs. The ay cae beconies ‘the Standard against which 
actual performance ies npirts: processes, and ,outputs) is compared. The 
decisions “that principals and feachers make in initial” program specification ‘ 

. May. subsequently have’ to be’ mddified in the Light of performance datas Never- — 
‘theless, with a. clear definiy ion Of, what the program should be, movement for= 


-ward gan atvys be planned and made} all involved can 1 Press toward’ planned 


Ve 
program goals: The PAM apy roach sets things up in complete * program definition . 


e 


Storms for instructional eee makers, ~~ 


wher’ tine’ is, short, a] PROCESS ‘compiex, and evaluative data scanty 
. 


. vattonal timely, and effective decistonaaking is hard ‘to come ‘bye This - 


describes the daily life of principals and eeachers, Boek make hundreds 
S oa8 y 

of, quick decisions alf day ‘Long. Neither have time to sit back and describe 
. r + 
where they are pera or even with individual scudeness 


“withowe such, plans descriptions, ratipnal rogram iaprosendne fades 
Pi P 


quickly, ‘Ty isa dase that instruétional prograns- be defined, hoyever, 
- x ing . 


nee for the present, “These descriptions or program spectficat ions are. 


Kéeping the ‘management principles and concepts 8£ Figures 1, 2, 35 : is 
and 4 in mind, it is possible to wigzese. that Ae programs be: 
defined, at a minimum, in terms of intended pr 


including: : Sree 


tr 


2s 


‘4, 


5s 


6. 


7. 


intended curriculums “c 


level, grade level, or individual student level (note that these 


igram and learning ‘outcomes 5 


apprppriate designation and use of available physical resources; 


specification of intended learning olitcomes for major content ‘ * 
areas like math, reading, and socia ‘studiese-that is; 
cufriculum specification; : . 


appropriate sélection and use of £ tive and summative 
tests derived from curricula and uged ‘correctively at the 
program.and teacher level;" 


teaching of content that is derived from the,curricula, 14 
that is, being sure that taachers are emphasizing the 


appropriate use of quality teaching tecHniques like cueing ~ . é 
(ives, defining what is to be learned a's well as giving 

directions as to what the learner should be doing during. the 

learning process), reinforcement, and encouragement of 

participation overtly or covertly th the learning acts 

appropriate allocation and use of instructional time under 3 
relevant teachingslearning conditions. ; ae 
standards of student achievement for major curriculum 2288 ' 
areas (@eg., reading, math, and writing) e-set at the program 


standards of achievement could be correlated with ‘os identical . | 
to those of the State or Regents Competehey Program); 


knowledge of the students! skills (iver, cognitive entry. 
characteristics) in relation.to the curriculum (iees, content 
and skills )«'"Cognitive entry characteristics are, highly . 
alterable because they represent particular content amd ~ ° 
skills which may be learned if they are absent, which may ;. | 
be reviewed if they have been forgotten, ‘and which may be’ 

learned to 4 criterion‘level if they have-been learned to a. 
lesstr level." (Lbid., MESA SEMINAR,, 19803. a3 é 


basic knowledge of the student's home environment and. “4. & 

parental status; r ft 
. © i, 

basic knowledge of teachér and student attitudes ” toward 

instructional Programs as defined; » : 78 Bae 


and services. : P = 


= - par . 


‘Because Little program definition, evaluation, and development occur 
Locally,- ic is “imperative that strategies. and techniques be devel oped hae. 
will. support local Program planning and dévelopment activities. 

. Techni ques that improve the management of programs and instruction 
usually do so by cihiccuetas programs, assessing performance, and returning 


evaluative data co principals and teachers. These technologies handle: 


decision=specific data (evaluative) in efficient ways with helpful con» 


figuyations and displays such that informed decisions can be made by 
ey 

principals and teachers. | Techno logies that help support school decision 
making will help principals and teachers play their roles aS instructional 


leaders and managers more effectively and efficiently. There is then good 


reason to beLieve that suststped improvements in principals' or instruc= 
* 


tional managers' skills all euie the, acquisition and continuous use of 


appropriate decision-support technologies. No amount of 'mon-supported! 


dnservice or training can be substituted for training in the use of 


decision-support technologies which inherently improve the quality of 
: os ‘i ) 
instructional management through their continuous use. 


Proposed Solution : f 


: 4 . 
If our schools are to reverse the recent trend of diglinishing pupil 


achievemont, then educational programs must be effectivély managed. Programs 


must be defined, desired processes and levels of achiévemenct must be 


I 


stipulated, and program evaluation must be carried fut. As indicated, 


principals are very busy, even harried. If they 


re to exercise adequate 


management control over their educational prog is, they must have manage 


aa ment tools. Title 1 PAM-=Program Analysis and Monitoringasis sf&h a . 


, 


managenént tool. *Presently available for vse in~the management of reading 
Programs, Title I PAM has the advantages/of being (1) useful, (2) timely, 


and (3) efficient. 


eo 3 : TImplicitly, program management requires infomation for decision 
oy mak ings Such information must be useful to the program managére In the | 


context of educational program management _ ipfo rmat ion is at acéater utility 


Y 
to the, ere that a is urelated to pupil achievement and to the extent 
wthat it As related to resources or conditi ns which the school can control. | 

. # 


For: exfinple, information on parental income may help shape a principal's ' 
Wi 2 b J 


/ ¢ ‘S 
expedfations regarding level of pupil achievements ‘but consider the relative 


» utility-sin program managenenti~-of auch information as compared, say,” to 
pe ee of hours per week which teachers devote to readihg instructions 
urs per week is a resourcé or ‘conditYon over which the school has Seat rol; 
schools cannot, however, control parental incomes Not all infoxmation 4 
: any f 


of equal usefulness, then, in educational program managements 


Information for “apr ional decision mak ing must’ also’be timely. | 
«The timeliness of information is a-function of its availability for 


decision mak ing Tf a pringipal receives information today but had to 


5 we, as : 
“make a decision yesterday, then o amount or degree of accuracy or 
pertinence will puke such, late information “useful. in the cqntext of 


“pract prone management, that information is most timely which maximizes 


“formative or ongoing program evaluation. This is, not to * Fiserace the 
ee a 7 . + i 


importance of summative evaluation, but ‘rather to Suggest that Program 
: 7 ees 
gars: requires formative evaluations If .a principal or program manager 


can devect problems in his ritfe’ ‘I reading program “at mid-year, hé may be 


able to take corrective action to avoid impending failure and achieve 


success. But summative evaluation does not permit on«course. corrections. 


Late or unavailable information is no more useful than nonexistent- informa 


tions : : e 


ds : ie é te Ay 4 
ey The practical question for program management is how to get useful 


and timely, information efficiently. An information delivery, process which . 
significantly impinged on a principal's or teacher's time would be an 

‘ ‘ ‘ 
inefficient processs, Briéfly, all other- things: being equal, the ifforma- 


»~tion-delivery method or process which requires the least amount of management 


and staff time is the miost efficient methods Many management systems on 


the markét are impractical to implement because much time is required °° 
. ’ 
either to understand them or’ to implement them, We have already indicated | = 
ms that time is a scarce commodity for, ptincipals. Similarly, one.hears that a 


money is scarce. Thus, “expehse is a factor in determin ing the practicability 
of an information delivery system. Ceteris paribus, the ease expensive 
method is the most advantageous. — ; ‘ 

, Title I PAM, as.stated above, has the properties of usefulness, 
timeliness, and efficiencys=in the ‘sense of minimum interference with the . 
educational program and modest cost. Minas, then, ds Title I PAM? | The 

: Title I PAM aston in reading comprises Acbievésene Menstars for the 

measurement of reading comprehens fons Monitor Reports on teachers! reading 


classes or instructional groups; Program Analysis Reports-=grade-level 


reports for the principal or program manager; questionnaires for the 


collection of process information; and computer programs for test scoring , *s ‘ 
and report generations . a, : i 
F The heart'of PAM is the Program Analysis Reporte This management tool 


is a grade lLevel*report which relates pupil achievement to the distribution 


. ae ot 

and use of classroom resourcese The basic structure of the PAM Program Analysis 
+ s + 2 . 

Report is illustrated in Figure 5. Thé juxtaposition of, program resources 

f ak o : 


, and pupil achievement facilitates management decision makings For example, if 


either pupil acHievement or the pattern of resource distribution and use 


“shown in ‘the ‘report do not meet the principal's éxpectations, he can direct 


his staff to alter program resource utilization and then observe, on ‘the 


next report, generation, whether~changes have occurred in program processes 


or pupil ‘achievement. If the question is raised, How doe the principal, 
i that the configuration of tesouieed and time depicted i the Programi 
Analysis Report matches actuality, the reply is that this question illustrates 
one of the strongest features of PAMe The reliability of reported resource 


distribution and use, in the PAM Program Analysis Report is, assumegs If pie 
‘a. * 


. t 


dehiévenene is substandard, then the program manager has a legitimate ead 
“to intervene at the level of the clas<room teachérs If pupil achievement 


and resource utilization are both pat isfactory.s there is no need for principal 


antenvention at the teacher levels However, satisfactory “pupil , achievement 


coupled with “uneimected resource utilization may wérrant intervention, le. 


popil achievement is canard but, say, twice the expected amount of 
e 


time is devoted to instruction, then a ‘problem of instructional inefficiency 


is apparente On the other hand, tf pupil achievement is satisfactory bit 


“resource uttlization is 50 percent less than expected, it is time for the 
principal to confer.with the teacher who is such a model ‘of efficiency and 
productivity, in order to learn whether the cause of this productivity is 
some teaching technique which may be transferable to other teachers. 
In Title I PAM there ave thie variations on the Program Analysis Reports 

* 


the ‘Type 1 report, the Type Il report, and the typStn reports Descriptions 


of each of these three reports follows ” 


The Title 1 PAM: Type 1 BuildingsLevel Reading Report, one df- four, 


sections of the Type I report, is based’on the infomation on Title i 


“ programs supplied by school districts té the New fork. State Education 
‘ e. 


- ee 
- Deparenegte These data are storéd on computer tapes Ccallgli LEAPssLocal * 
. ‘* Ff 
Educational Agency Prograns-=-tapes) and accessed by, the PAM Type 1 Seompatsr 
' , 
program, The Type 1 Yeport, then, contains infomattone-puptls, norm 


referenced test ‘information, instrugpdonal activities, “contact hours, Pere ‘ 


cent ‘of pupils with limited Enghish ptoficiency, and pprcene of ‘disadvantaged 


pupils-erequired by the State om cavagorical lyn! programs, ‘As 


i iljustrated | in Figure 6; thé Type I-report varies * format from the basic 
PAM: Program satus Report Structure only in chad! pupil achievement is me 
the first variable reported. — 

+ 
. The Type 1 report. is based on Summative evaluation data, and its, 


utility lies in its facilitation of summative eval vat ton through a useful 


Sacaaa of Program resources and outcomes. VAs a histoyicalenote, ‘he . 
> 


++ Type L deport’ sofewage (developed in 1980 idee: a Federal Refinements: : 
‘ 
contract) has been used" to} produce reports vaich were the basis of the : 
cy ” 
state's Title’ I report to the ‘Federal governnent ‘these’ “reports are also 


tbe fg sent to all fy State school districts serving ‘Title I ecudente:s and 
. ‘ 


other catego rically-aided students, The cost of gerierating.the Type 1 


report for $00,000 student's was approximately $.02 (two cepts) per student. 
. bs # ie 


Affordifig 2 more specific and detailed ‘analysis of the Title I reading 


program ehrough its focus on the level of the instructional group: is the 


Title 1 PAM Type Il Program’ Analysis Report, one of four sections of the” 


Type IL Report. The @iformation in the Fype 12 Program Analysts Report 
. ‘ 
(Figure 7) includes pupil achievement, number of pupils and pupils’ receiving 


stednchetapeateerer ins instruct ional prograns, . sana series of time 


variablesenallocaged’ time, instructional coordét Qn time, and time in 
3 


instructional setting and by. instruct onal enphas isceal enpressed in, 


¥ minutes per week. The particular utility of the Type Il Progran analysis 


* . ‘ : 
i PN 5 Sis pote - 
Report lies in the special ynaturt of the pupil achievement informatign it 
: oi wc of te ¢ NR a % S . 


+ contains and*in the de 


» major strengthis’ of the monitors, are aia construct, validity as paraphrasee 


\ 


based theasures ae resting) comprehension; their multiple levels of 


‘teow grade, fay at their sealing P copert ies. Suits perfomanceson all 


aifficulty, | thus, readily permitting comparisons between classes at a grade 


‘level, across grade levels, and from year to aes g ere 


The most edapriiens ive variation on the PAM rogran Analysis Report: 
is . the tide I PAM Type, Ill-report. The. assumption eae this report is 
that local evaluation of tee programs shoiild “ocgur inthe context of 
the, total instructional progran. The Type Ty Report ts a management topl 


® for compréhensivé. program anaiysis and “evaluat ions As illustrated in 


‘ s 


_ Figures 8 ee 9, the Tipe, HTT report enjoys \the advantages of additional: 


dimensions and more extensive “information than does the Type ut Féporte : 

° THe Type Til ‘Program Wiatysie Rept artalyzes the total fending program’ 
a classes ata grade devel. “Within each class, students receiving, supplenen« 
tal Titye I ins ttuction’ are differentiated “from those nat recgivin! sich | 


* suppldnental instruct{on. Additionally, as Viguve 8 shows, the, Type IIl 


4 report gives information on onentorone instruction, supplemental instruc= 
: ‘ gion, and teacher experiences . : ad ., : . 
Figure 9 illustrates the mule tginend tone, anialyets of the clagstoom 
instructional’ sett ing which is” é “antique: feature’ of the Type ITI Program 
aie 


% ’ i : . . . 
Analysis Reporte ‘The variables displayed in this section of the report 


include time in whole class and small. group instruct ion; time in four types 
: + . 1 
soe r+ , - 
of instructional configurations within whole class and small groupeedirect 


instruction from teacher, other adult working with pupils, pupils working 


independently, and pupils working in groups without an adult; instructional 
_ groupe ranked by ability; group size; and Title I and non-Title I students 
} % . Cone 2 
within groupse This section of the Type III report offers a very compres 


hens ive picture of reading instruction at a grade levél. < N 


: while comprehensive in itself, the Type III Report was designed to be 

used in conjunction with the Type II Report to give a program manager ‘a 

compiete information system to facilitate dacis ion makings O£ course the is 

Type 1 Report may also be igea together with the (Type IL and Type IIL 

reports, for a total formative and ‘summative evaluation systeme 

tg 18" should be noted in conclusion that the Type III Report contains 

’ ad - . several other sections in addition to those ‘illustrated and described in « 
the texte, These additional sections, along with the sectifons of the Typg,1 
and II reports not described abogg, are illuserated’ in the Appendix.”. : 


ik 
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« Policy and sion and Goals cation and 7 
Gontro) Gentral Adminis. 
. { 2 7 tration 
é fas : 
2, ‘Educational Intended Program 
Programming . Out@omes (Pros - istration 
‘ gram Def inition) 
3 is 
3,, Instructional Intended Learn+ 


‘ », Programming 


«3 


» Organizational < Longeterm Mise 


ing Outcomes 

* (Curriculum 

a Specification) 
% 


Support Pere 
sonnel 


Board of Edue ¢€---Aggregate ————> Board of Educa» 


Building Admin <-- Individual+——> Central Admins 


Instructional/ €--- Individual 


tion \ 
Range Planning 
4 


Evaluat ion,of 
Aggregate istration 
‘Reporting 


——> Building/Program 


Administration Reporting and 


‘ ‘ 
‘Instructional Plane 


. : _ ning 


Responsibility is based on authority (ises, the right to command to take or not take action) since it 
is the obligation to use authority in order to have work performed correctly (Cook, 1971, pe 22) 


Passessment data ( <=) on student needs and program rescurces provides‘ the basis for establishing 
Responsibilities, accurately placing students in specified curricula, and defining programs+ . 


“Evaluation data(€) on student achievement and program management performance provides info 
réquired t® demonstrate Accountability, to modify program implementation strategies, or to redefin 


programe 


ion 
he 


Se countabi lity” is, defined as the responsibility of reporting the extent to which one's other assigned . 
responsibilities are mets : F 


% 


z *Adapted from "A Model for the Deve lopment’ of 


a 


an \ 


FIGURE 1, SCHOOL DISTRICT MODEL OF PROGRAM DEFINITION, MANAGEMENT ROLES, AND DATA FLOW* 
. . 


Special Education Wanagement System for the Education 


of Children with flandicapping Conditions,” IEC Consortium, State Egucation Departmept, Albany, New York, 


March 1978, page 8. 
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